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ABSTRACT 



Florida's Workforce Development Education (WDE) Program was 
subjected to an evaluation and justification review to determine its benefit 
to the state and evaluate its systemwide performance. The following 
components of Florida's WDE program were evaluated: career and technical 
education (CTE) ; adult education programs; and continuing education programs. 
The review established that Florida's WDE program provides a public benefit 
and should continue. It was credited with meeting local and state workforce 
needs and helping Florida compete in a global economy. In fiscal year 
2000-2001, the WDE program met most standards for the number and percentage 
of associate in science degrees and college credit certificate program 
completers placed in jobs or continuing their education. The program's 
performance for its remaining legislative measures could not be evaluated for 
several reasons. Associate in science degree program completion rates tended 
to be low; however, CTE programs generally improved program completers' 
earnings. Nine exhibits are included. The following items are appended: 
Florida requirements for justification reviews; descriptions of the 
activities of the three divisions assigned WDE program responsibilities; 
descriptions of Florida's career and technical education, adult education, 
and continuing workforce education programs; earnings over time for students 
in the 1995-1996 cohort; and the Florida Department of Education's response 
to the study. (MN) 
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Executive Summary ■ 

Justification Review of the ; 
Workforce Development Education 

Purpose — — — — — 




This report is one of three reports that present the results of our program 
evaluation and justification review of the Workforce Development 
Education Program administered by the Department of Education.) This 
report evaluates' the Workforce Development Education Program's 
benefit to the state, workforce development education system-wide 
performance, and program performance measures. A second companion 
report will concentrate on the performance of adult education/literacy 
programs. A third report will evaluate performance of apprenticeship 
programs. Together these three reports cover the statutory; requirements ' 
for a justification review. • , - * ' 



Introduction 



The purpose of the Workforce Development Education Program is to help 
students attain those skills that enable them to become or remain 
economically self-sufficient. The program provides training designed to 
meet local and state workforce needs and to help Florida compete in a 
global economy by building a broadly based, highly skilled, more 
productive workforce: - ’ ... v \ 

T \ ^ 

The 28 community colleges and 58 of the 67 district school boards in > 
Florida carry out program service delivery. In Fiscal Year 2000-01, school 
district programs served 643,901 students and community colleges served v 
360,588 students in workforce programs. , 

\ ‘ v.. 

For ease of discussion, we have grouped Florida's public postsecondary 
workforce development education programs into three categories: career^ 
and technical education programs, adult education programs, and 
continuing education programs. Career and technical education ' 
programs prepare individuals for entry into a specific occupation by/ 
completing N 

■ an associate in science degree, ' . , 

■ a college credit certificate, 

, ( v t ' 

■ an adult vocational certificate, or • A , , 

■ an apprenticeship program. x 
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Executive Summary 



Adult education programs provide courses for individuals who need 
literacy, basic education, and English language training to improve job \ 
performance and/or to move into higher paying jobs. Continuing 
. ' education programs are courses designed to improve skills for individuals 
^ ; who are already employed. v 

v The program is funded predominantly by general revenue. For Fiscal 

Year 2001-02, the program was appropriated $796,342,765, of which 
’ $695,740,368 or 87% was from general revenue. Most (97%) of the general 
. . revenue funds are allocated to community colleges and school districts for 

program delivery, with the remainder to categorical grants. 

Workforce development funding is based partially on program outcomes 
(completions and placements). As required by s. 239.115(7)(a), Florida 
v Statutes, at least 15% of the funds for workforce development education 

programs are distributed based on performance outcomes. The 
remaining funding is based on the prior year's funding levels. 

. - ' / ^ ^ ' . . V . 

Program Benefit and Placement — - — — 

: n. 1 ■ 

The Workforce Development Education Program provides a public 
. benefit and should continue. The program helps individuals attain skills 

that enable them to become or remain economically self-sufficient. When 
' individuals become, self-sufficient, the state receives a financial benefit 

through increased tax revenues, reduced dependence upon public 
assistance, and reduced incidence of incarceration in the state's 
correctional system. The program provides training designed to meet 
local and state workforce needs and helps Florida compete in a global 

economy by building a more broadly based and productive workforce. 

: '• •• ,• > r 

. Abolishing this program would reduce the availability of career and 
technical education and adult education programs to the public. The \ 
Workforce Development Education Program is placed appropriately in 
the Department of Education. ^ 



Program Performance — — 

The program's performance in meeting Fiscal Year 2000-01 legislative 
performance standards can only be evaluated for associate in science 
\ degree and college credit certificate programs. The program met most 

- . standards for the number and percentage of associate in science degree 

and college credit certificate program completers placed in jobs or 
continuing their education. We could not evaluate the program's , 
performance for its remaining legislative measures for three reasons: 

(1) performance data for measures assessing adult vocational certificate 
programs is not comparable to data for prior years due to the planned 
phase-in of performance funding over time, (2) baseline data to establish 
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• . • ^ , * . 1 , ' ’ V Executive Summary 

reasonable standards has only recently become available for the measure 
assessing adult education programs, and (3) the department has not 
completed data collection needed to establish performance and standards 
for measures relating to national accreditation. , 

To supplement the limited, information currently provided by the 
program' s' legislative performance measures, we further evaluated the 
program's effectiveness using other data and found that participants in 
associate in science degree programs tend to have low completion rates. 
Nonetheless, career and technical education programs'in general improve 
the earnings of those who complete them. •. N 

"i \ 

The program's legislative performance measures will provide some useful 
information once the effects of the planned phase-in of performance 
funding no longer prevent performance comparison across years. 
However, revisions to the measures are needed to better evaluate 
program performance. To improve the usefulness of the program's 
legislative performance measures in evaluating program performance, the 
Legislature should consider revising the program's measures to focus on 
higher levels of post-completion earnings, include completion rates for 
career and technical education programs, and include an additional 
outcome measure for adult basic education. '• 



Agency Response 

.C N ' ■ 



The Commissioner of Education and the Secretary of the. Department of 
Education provided a written response to our preliminary and tentative 
findings :and recommendations. (See Appendix G, page 38 for their ) 
response;) A 
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Chapter 1 

Introduction 

Purpose 



This report is one of three reports that present the results of our program 
evaluation and justification review of the Workforce Development 
Education Program administered by the Department of Education. The 
28 community colleges and 58 of the 67 district school boards in Florida 
carry out program service delivery. 1 The 1994 Government Performance 
and Accountability Act directs the Office of Program Policy Analysis and 
Government Accountability (OPPAGA) to conduct justification reviews of 
each state program during its second year of operation under a 
performance-based budget. Justification reviews assess agency 
performance measures, evaluate program performance, and identify 
policy alternatives for improving services and reducing costs. 

This report evaluates the Workforce Development Education Program's 
benefit to the state, workforce development education system-wide 
performance, and program performance measures. A second companion 
report will concentrate on the performance of adult education/literacy 
programs. A third report will evaluate performance of apprenticeship 
programs. Together these three reports cover the requirements for a 
justification review. See Appendix A for a summary of our conclusions 
about this program based on the statutory requirements for justification 
reviews. 



1 The Department of Education's website provides the location of the state's community colleges and 
school district postsecondary voc-tech centers. See http://www.dcc.fim.edu/coneges.htm for the location 
of the community colleges and http://www.fim.edu/doe/workforce/vtc3 95.htm for the location of the voc- 
tech centers. 
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Introduction 



Background 

The purpose of the Workforce Development Education Program is to help 
students attain those skills that enable them to become or remain 
economically self-sufficient. The program is funded predominantly by 
general revenue. For Fiscal Year 2001-02, the program was appropriated 
$796,342,765, of which $695,740,368 or 87% was from general revenue. 

The program provides training designed to meet local and state workforce 
needs and to help Florida compete in a global economy by building a 
broadly based, highly skilled, more productive workforce. Postsecondary 
workforce education programs are an integral part of the state's system of 
public education and essential components of Florida's K-20 education 
system. 

Program structure 

The governance of Florida's public education system is undergoing a 
major reorganization. The 2001 Legislature passed the Florida Education 
Governance Reorganization Implementation Act. This act sets out the 
process by which the K-12, community college, and state university 
systems will be merged into a seamless K-20 system. Under this act, the 
Florida Board of Education was created effective July 1, 2001, and will 
oversee the entire K-20 education system. The current State Board of 
Education retains the power to override decisions of the board until 
January 2003, when it is dissolved. (See Exhibit 1.) 
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Exhibit 1 

Transitional Organizational Structure for Florida's K-20 Educational System, 
Effective July 2001 

FLORIDIANS 




Source: Chapter 2001-170, Laws of Florida, the Florida Education Governance Reorganization 
Implementation Act 



Currently, Workforce Development Education Program oversight 
responsibilities are assigned to three divisions within the Department of 
Education: the Division of Workforce Development, the Division of 
Community Colleges, and the Division of Technology. See Appendix B 
for more information on the activities of the three divisions assigned 
workforce development education responsibilities. 

Partnerships with other workforce programs in Florida. The Workforce 
Development Education program works collaboratively with other 
economic development and workforce programs in Florida. State 
administrators serve on statewide boards and committees that address the 
state's workforce needs, such as the Council of 100 and Workforce Florida, 
Inc. The Division of Community Colleges and the Division of Workforce 
Development collaborate with other state level entities to address 
workforce development needs. For example, Operation Paycheck is a 
collaborative effort between the state education system and the state 
workforce system to develop a rapid response re-employment effort for 
workers dislocated due to the September 11 terrorist attacks on our 
country. 
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Local administrators engage in similar partnerships to address local 
workforce development needs. For example, the law requires community 
college presidents and school district superintendents to serve on 
workforce boards and on high wage/high skills jobs committees. Local 
community colleges and school districts are key components of the one- 
stop delivery system, which is the state's primary customer service 
strategy for offering every Floridian access to services such as job search, 
referral, and placement assistance; career counseling; and support 
services, including child care and transportation assistance to gain 
employment. The one-stop centers contract with community colleges and 
school districts for services, and many of the one-stop centers are located 
in community college and school district facilities. 

Program delivery 

Workforce development education programs are provided by 58 of the 67 
school districts and all 28 community colleges. In Fiscal Year 2000-01, 
school district programs served 643,901 students, and community colleges 
served 360,588 students in workforce programs. 

For ease of discussion, we have grouped Florida's public postsecondary 
workforce development education programs into three categories: career 
and technical education programs, adult education programs, and 
continuing education programs. 

Career and technical education programs prepare individuals for entry 
into a specific occupation by completing 

■ an associate in science degree, 

■ a college credit certificate, 

■ an adult vocational certificate, or 

■ an apprenticeship program. 

Community colleges provide all of the above career and technical 
education programs, and school districts provide adult vocational 
certificate and apprenticeship programs. See Appendix C for a more 
detailed description of the various career and technical education 
programs. 

Adult education programs provide courses for individuals who need 
literacy, basic education, and English language training to obtain jobs or 
move into higher paying jobs. Appendix D provides a more detailed 
description of adult education programs. 

Continuing education programs are courses designed to improve skills for 
individuals who are employed already. See Appendix E for a description 
of continuing education programs. 
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Introduction 



Program funding 

The 2001 Legislature provided $796,342,765 for the Workforce 
Development Education Program, with $100,602,397 from trust funds and 
the remainder from general revenue. Most (97%) of the general revenue 
funds are allocated to community colleges and school districts for 
program delivery, with the remainder to categorical grants. (See 
Exhibit 2.) 

Exhibit 2 

Workforce Development Education Program Primarily Funded 
Through General Revenue 
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Program Delivery 






Community College 
Workforce Development Programs 


$292,923,869 




School District Workforce Development Programs 


379,318,081 




OtherFunding 






Critical Jobs Initiative 


$ 4,989,987 




Workforce Education Programs for 
Adults with Disabilities 


18,508,431 




Adult Basic Education federal flow-through funds 




$ 23,457,545 


Vocational Formula Funds 




77,144,852 


^Ot^^ipropriaiibns r ; 


^;740i368 : 

1* 





Source: Chapters 2001-253 and 2001-367, Laws of Florida. 



The Division of Workforce Development and the Division of Community 
Colleges perform the Department of Education's oversight functions for 
the Workforce Development Education Program. These divisions were 
appropriated $7,741,491 and $4,704,989, respectively, to conduct their 
activities in Fiscal Year 2001-02. In addition to workforce programs, the 
divisions' activities include providing oversight for secondary vocational 
programs and associate in art degree programs. 
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Performance Funding. Workforce development funding is based partially 
on program outcomes (completions and placements). As required by 
s. 239.115(7)(a), Florida Statutes, at least 15% of the funds for workforce 
development education programs are distributed based on performance 
outcomes. The remaining funding is based on the prior year's funding 
levels. For Fiscal Year 2001-02, $93,059,669 (15%) of the program's 
appropriation for program delivery was distributed to community 
colleges and school districts based on performance. 2 

As discussed in OPPAGA's November 2001 Program Review of the 
Workforce Development Education Program, the performance funding 
system has had a positive impact on the workforce development 
education programs. 3 Because the funding process is driven by program 
completion and job placement, the community colleges and school 
districts that provide workforce education programs have restructured 
their curricula to offer more programs that meet labor market needs and 
to eliminate poorly performing programs. Examples of actions taken by 
local program administrators to improve program performance include 
providing guaranteed employment for certain program completers, 
working with businesses to increase use of internships, and educating 
administrators and instructors on how performance affects funding. 



2 Funding of $56,992,520 for continuing workforce education programs is excluded from calculations 
for performance funding. 

3 Program Review, Workforce Development Education Program > Florida Department of Education, 
OPPAGA Report No. 01-56, November 2001. 
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Chapter 2 



Program Benefit and Placement 



The Workforce Development Education Program provides a public 
benefit and should continue. The program helps individuals attain skills 
that enable them to become or remain economically self-sufficient. When 
individuals become self-sufficient, the state receives a financial benefit 
through increased tax revenues, reduced dependence upon public 
assistance, and reduced incidence of incarceration in the state's 
correctional system. The program provides training designed to meet 
local and state workforce needs and helps Florida compete in a global 
economy by building a more broadly based and productive workforce. 
Abolishing this program would reduce the availability of career and 
technical education and adult education programs to the public. The 
Workforce Development Education Program is appropriately placed in 
the Department of Education. 

The program provides a public benefit 
and should continue 



Workforce program 
completers earn more 
than high school 
graduates 



The Workforce Development Education Program helps individuals attain 
skills that enable them to become or remain economically self-sufficient. 
Attaining this goal helps Florida compete in the global economy. Building 
a broadly based, highly skilled, more productive workforce contributes to 
Florida's economic prosperity. 

Workforce development education programs lead to higher earnings. An 
analysis of Fiscal Year 1999-00 program participants shows that vocational 
certificate and associate in science degree program completers earn more 
than high school graduates. (See Exhibit 3.) Annualized earnings for 
program participants with full-time employment ranged from $25,968 for 
those who completed school district adult vocational certificate programs 
to $33,536 for those who completed an associate in science degree. High 
school graduates with full-time employment earned an annualized 
average of $16,164, substantially less than workforce program completers. 
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Program Benefit and Placement 



Exhibit 3 

Annualized Earnings for 1999-00 Program Completers 1 




1 Based on October to December 2000 earnings data for individuals who completed programs in 
1999-00 and were employed full-time October through December 2000. 

Source: OPPAGA analysis of data from the Department of Education, Florida Education Training and 
Placement Information System. 



Increased earnings benefit individuals completing workforce 
development education programs and benefit the state through 

■ increased tax revenues; 

■ increased economic activity and growth; 

■ decreased crime and resultant increased incarceration rates; and 

■ decreased reliance upon public financial support . 4 



4 Numerous sources discuss the benefits of workforce development education programs. For 
example, see "Remarks Prepared for Delivery by U.S. Assistant Secretary David Sampson/ National 
Governors Association, Workforce Development Policy Forum, 2001; "A New Approach to Workforce 
Development," presentation prepared by Workforce Florida Inc., Workforce Development Policy 
Forum, 2001; "Education and Investment in Human Capital," Journal of Political Economy, Burton A. 
Weisbrod, 1962; "Education and Economic Growth," Social Forces Influencing American Education, 
16 th Yearbook of the National Society of Education, Theodore Schultz, 1961; and "High School 
Dropout Costs," Jay Pfeiffer, Workforce Education and Outcome Information Services, and David 
Wright, Council for Education Policy Research and Improvement, October 17, 2001. 
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Program Benefit and Placement 



For example, a recent study illustrates the positive influence of the 
Workforce Development Education Program's various adult education 
programs on tax revenues. This study estimated that if all Florida high 
school dropouts were to attain a high school diploma, the annual gain in 
state sales tax revenues would be $769 million. 5 The estimated effect on 
federal income taxes would be even higher — an annual increase of 
$2.3 billion. These gains would accrue due to the increased earnings 
potential of Florida's citizens. 



Program helps meet 
demand for highly 
skilled workers and 
future labor market 
needs 



The Workforce Development Education Program is designed to respond 
to Florida's workforce needs. Forty-six percent of the 46,121 individuals 
who obtained job placements after completing state-supported career and 
technical education programs in 1998-99 were placed in occupations 
designated as high wage/high skill in 1999-00 by the Department of 
Education. 6 



In providing training programs, the Workforce Development Education 
Program also helps Florida meet future labor market demands. The 
program provides training for 16 of the 20 fastest growing occupations in 
Florida. (See Exhibit 4.) The program also helps meet demands for other 
rapidly growing occupations. Employer preference for education and 
training requirements vary depending on labor market demands. 



5 "High School Dropout Costs," Jay Pfeiffer, Workforce Education and Outcome Information Services, 
and David Wright, Council for Education Policy Research and Improvement, October 17, 2001. 

6 Based on information obtained from the Department of Education's Workforce Education and 
Outcome Information Services (WEOIS). This information was generated by the Workforce 
Estimating Conference and Workforce Florida, Inc., which develop targeted occupation lists. The lists 
include high wage/high skill occupations needed both statewide and within each of the state's 24 
workforce development regions. 
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Program Benefit and Placement 



Exhibit 4 

The Program Provides Training for 16 of the 20 Fastest Growing Occupations in Florida 



Fastest Growing Jobs in Florida/Hourly Wages 



Annual Percentage Increase 



COMPUTER SUPPORT SPECIALIST ($18.40) 

SYSTEMS ANALYST ($24.96) 

Computer Engineer ($29.39) 

SURGICAL TECHNICIAN ($12.75) 

PARALEGAL ($17.00) 

Instructional Coordinator ($19.23) 

DATABASE ADMINISTRATOR ($23.25) 

MEDICAL ASSISTANT ($10.53) 

PACKAGING AND FILLING, MACHINE OPERATOR ($8.50) 

MEDICAL RECORDS TECHNICIAN ($9.95) 

PHYSICIAN ASSISTANT ($24.09)’ 

RESPIRATORY THERAPIST ($17.54) 

HUMAN SERVICES WORKER ($11.50) 
TELEMARKETER, DOOR-TO-DOOR SALES, STREET VENDING 
($1 0.88) 2 

CORRECTIONAL OFFICER AND JAILER ($13.75) 
HOME HEALTH AIDE ($8.30) 
ADJUSTER CLERK ($10.90) 
Securities, Financial Service Sales ($31.67) 
PRODUCER. DIRECTOR, ENTERTAINER ($16.42) 
Engineering, Science, Computer Systems Manager ($31.84) 



Occupations for which the Workforce Development Education Program provides training 
Occupations for which the Workforce Development Education Program does not provide training 




10 . 6 % 



1 One community college (Miami-Dade) provides physician assistant training. 

2 The program provides training for customer service representatives, which encompasses the skills used in telemarketing. The 
program does not provide training for door-to-door sales or street vending. The categories used in this analysis were established by 
the Agency for Workforce Innovation for the purpose of reporting labor market statistics. 

Source: Agency for Workforce Innovation, Office of Workforce Information Services, Labor Market Statistics. 
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Program Benefit and Placement 



Abolishing state 
funding would reduce 
the availability of career 
and technical education 
and adult education 
programs 



The Workforce Development Education Program provides benefits to 
both the individuals that attend these programs and to the general public. 
To achieve these benefits, potential students must have access to a 
workforce development education program. By providing a system of 
public postsecondary workforce education, the state lowers the cost and 
increases the accessibility of career and technical education, adult 
education, and continuing workforce education programs. 



Access to workforce development education programs is particularly 
important in Florida because 77% of Florida's adult citizens do not have a 
college degree. Moreover, only 56% of high school freshmen in Florida 
graduate within four years. Workforce development education programs 
are specifically targeted to serve these citizens. 



If state funding for workforce development education programs were 
discontinued, the likely short-term impact would be a reduction in the 
availability of career and technical education and adult education 
programs for the general public and higher costs for program participants. 
The absence of state funding would force community colleges either to 
increase tuition and fees and pass on more of these costs to students or 
drastically decrease the number of career and technical education 
programs available to the public. Some students may not be able to afford 
tuition increases. For example, a course that now costs about $50 per 
credit hour would increase to $200 per credit hour. 7 School districts may 
have to eliminate career and technical education and adult education 
programs due to resource constraints and needing to focus on providing 
K-12 education services. The long-term effect of fewer career and 
technical education and adult education programs would be fewer 
trained persons for Florida's job market, an increased reliance on public 
assistance, and a higher state incarceration rate. 



Two types of entities might take over some of the program's 
responsibilities if state workforce development education funding was 
discontinued, but neither would be in a position to fully meet the service 
demands being met by the program. 

■ Private institutions currently compete with community colleges and 
school districts for participants in the career and technical education 
arena, and would likely be interested in receiving more of this 
business. However, their interest may be limited to increasing their 
training offerings for high demand, high wage jobs for which they 
would be able to make immediate placements and charge higher 
tuition. Private institutions are not likely to be capable of meeting the 
total state demand for other types of career and technical education 
programs. 



7 Florida law requires student fees to cover approximately 25% of the program costs for Florida 
residents and 100% of the program costs for non-residents. This estimate is based on fees currently 
charged by workforce development programs for resident and non-resident students. 



Program Benefit and Placement 



• Local workforce development boards currently provide a small 
portion of the state's adult education programs using federal 
Workforce Investment Act funds. These programs primarily are 
intended to help economically disadvantaged and unemployed 
persons become or remain employed. However, it would be costly for 
these boards to put the systems in place that would be needed to 
replicate the level of service and community outreach provided by the 
program. Because the target population served by adult education 
programs is difficult to reach, local community colleges and school 
districts have established extensive outreach programs to bring adult 
education services to participants in places such as churches, libraries, 
and hospitals. It is unlikely that local boards would be able to meet 
the needs of more than a fraction of the 500,000 people currently being 
served by school district and community college adult education 
programs. 

Program Placement 

The Workforce Development Education Program is placed appropriately 
within the Department of Education. The department has statewide 
responsibility for the K-20 public education system. One of the major 
goals of the K-20 system is contributing to Florida's economy through 
providing a skilled workforce. Improving the employability of 
individuals begins with a basic education and continues on through high 
school and on to higher education and employment. 
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Program Performance 

The Workforce Development Education Program's legislative 
performance measures can be used only to evaluate the performance of 
associate in science and college credit certificate programs. Associate in 
science/college credit certificate programs were effective in providing 
program completers with the training and skills needed to obtain higher 
wage jobs . 8 The program's other legislative performance measures 
cannot be used to evaluate performance due to program failure to collect 
data for measures relating to national accreditation and year-to-year 
changes in data resulting from the planned phase-in of the performance 
funding system. Also, because the Department of Education's inspector 
general has not completed an assessment of the validity and reliability of 
the program's performance measures, our discussion of performance 
based on the program's legislative measures is provided without 
assurance that the measures are valid or that the associated data is 
accurate. 

To supplement the limited information currently provided by the 
program's legislative performance measures, we further evaluated the 
program's effectiveness using other data and found that participants in 
associate in science degree programs tend to have low completion rates. 
Nonetheless, career and technical education programs in general improve 
the earnings of those who complete them. 

The program's legislative performance measures will provide some useful 
information once the effects of the planned phase-in of performance 
funding no longer prevent performance comparison across years. 
However, revisions to the measures are needed to better evaluate 
program performance. The legislative measures do not sufficiently 
emphasize higher levels of post-completion earnings, and do not include 
completion rates or adequate outcome measures for adult basic education 
programs. 



g 

In this instance the term "high wage" refers to the high wage/high skill occupations identified by the 
Workforce Estimating Conference and Workforce Florida, Inc., i.e., those for which program 
completers earn at least $9 an hour. Wages of $9 an hour would equate to $18,720 annually for 
full-time employment. 
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Associate in science/college credit certificate programs 
met most standards; comparable data not yet available 
for other programs 

Program measures The Workforce Development Education Program's primary outcome 
emphasize job measures show the extent to which program participants achieve certain 

placement levels of job placement after graduation (see Exhibit 5). These measures' 

performance standards require placements at three specific earning levels: 
Level HI, Level E, and Level I. 

■ Level IE is the most desirable of the three placement levels. A Level IE 
placement means that the participant completed a program for a high 
wage occupation and was earning $4,680 or more per quarter. An 
individual would need to earn at least $9 an hour and work a 40-hour 
week each week of the quarter to qualify as having a Level IE 
placement. 

■ Level E means that the participant completed a program identified for 
new entrants and was earning $3,900 or higher per quarter or was 
continuing his or her education at the college level. An individual 
would need to earn at least $7.50 an hour and work a 40-hour week 
each week of the quarter to qualify as having a Level E placement. 

■ Level I means that the participant did not achieve a Level E or IE 
placement, but was employed (regardless of income), in the military, 
or continuing his or her education at the vocational certificate level. 9 

The Workforce Development Program's measures report on performance 
for persons that completed programs and found job placements in prior 
years. For example, Fiscal Year 1999-00 measures represent persons who 
completed programs in 1997-98 and were placed as of 1998-99. Fiscal Year 
2000-01 measures represent persons who completed programs in 1998-99 
and received job placements as of 1999-00. 



Department staff uses Florida Education and Training Placement Information Program data to 
calculate results for these measures. If program completers are not found in a Level I, II, or III 
placement, this means that they were not found employed or continuing their education in Florida. 
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Exhibit5 

Most Performance Standards Met for Associate in Science/College Credit Certificate Programs, 
But Data Problems Preclude Analysis Using Measures for Other Types of Programs 







ige^ritaQej 



Performance amounts for Fiscal Year 2000-01 are in boldface type and shaded 
when standards were metbasedon a vaiidcomparison of performance data and standards. 



Adult Vocational Certificate Programs 


Number/percentage of vocational certificate program completers who 
are found placed according to the following definitions: 

(1) Level III - Completed a program identified as high wage/high skill on 
the Occupational Forecasting Conference list and found employed at 
$4,680 or more per quarter 


9,988 


33.4% 


13,772' 


29.5%’ 


12,227 


42.6% 


(II) Level II - Completed a program identified for new entrants on the 
Occupational Forecasting Conference list and found employed at 
$3,900 or more per quarter, or found continuing education in a college 
credit-level program 


2,975 


10.0% 


5,282' 


11.3%’ 


4,369 


15.2% 


(II!) Level 1 - Completed any program not included in Levels II or III and 
found employed, enlisted in the military, or continuing their education 
at the vocational certificate level 


12,583 


42.1% 


16,617' 


35.6%’ 


10,801 


37-6% 


Associate in Science Degree and College-Credit Certificate Programs 


Number/percentage of associate in science degree and college-credit 
certificate program completers who are found placed according to the 
following definitions: 














(1) Level III - Completed a program identified as high wage/high skill on 
the Occupational Forecasting Conference list and found employed at 
$4,680 or more per quarter 


7,161 


60.1%:/' 


7,310 


162L1% 


6,897 


57.9% 


(II) Level II - Completed a program identified for new entrants on the 
Occupational Forecasting Conference list and found employed at 
$3,900 or more per quarter, or found continuing education in a 
college-credit-level program 


1,402 


11.8% 


1,168 


9.9% 


1,351 


11.3% 


(III) Level 1 - Completed any program not included in Levels II or HI and 
found employed, enlisted in the military, or continuing their education 
at the vocational certificate level 


2,086 


17.5% ^ 


1,972 


.; 16.7% 


1,661 


13.9% 


Adult Education Programs ; 


Number of adult education, including English as a Second Language, 
and adult secondary education completion point completers who are 
found employed or continuing their education 


Not 

Available 2 


N/A 


42,21 8 3 . 


N/A 


57,344 3 


N/A 


Nationally Recognized Accrediting Standards ! 


Number/percentage of workforce development programs which meet 
or exceed nationally recognized accrediting standards for those 
programs which teach a subject matter for which there is a nationally 
recognized accrediting body 


Not Available 4 


Not Available 4 


To be developed 4 


Number/percentage of students attending workforce development 
programs which meet or exceed nationally recognized accrediting 
standards 


Not Available 4 


Not Available 4 


To be developed 4 


Number/percentage of students completing workforce development 
programs which meet or exceed nationally recognized accrediting 
standards 


Not Available 4 


Not Available 4 


To be developed 4 



prior year data or standards. 

2 The Department of Education does not have sufficient data to calculate Fiscal Year 1999-00 results for adult education programs. 

3 The standard was based on inadequate baseline data and thus the standard and actual performance results are not comparable. 

4 Despite being required by the Fiscal Year 2000-01 and 2001-02 general appropriations acts to collect this data, the department did 
not begin a data collection effort until fall 2001. Department staff expects results by mid-June 2002. 

Source: Chapter 2000-171, Laws of Florida, and Department of Education documents and staff. 
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The program's performance in meeting Fiscal Year 2000-01 legislative 
performance standards can be evaluated only for associate in science 
degree and college credit certificate programs. As shown in Exhibit 5, the 
program met most standards for the number and percentage of associate 
in science degree and college credit certificate program completers placed 
in jobs or continuing their education. We could not evaluate the 
program's performance for the remaining measures for three reasons: 

(1) performance data for the measures assessing adult vocational 
certificate programs is not comparable to data for prior years due to the 
planned phase-in of performance funding over time; (2) baseline data to 
establish reasonable standards has become available only recently for the 
measure assessing adult education programs; and (3) the department has 
not completed data collection needed to establish performance and 
standards for the measures relating to national accreditation. 



Most standards met for 
associate in science/ 
college credit certificate 
programs 



The program met four of six standards for the number and percentage of 
associate in science degree/ college credit certificate program completers 
found employed or continuing their educations. These programs 
performed well in achieving objectives to place a high number and 
percentage of program completers in the three job placement categories 
(Levels I, II and HI). The programs met standards for two of the three job 
categories and, more importantly, exceeded standards for the number and 
percentage of completers placed in higher wage jobs (Level HI). These 
programs also improved in placing program completers in higher wage 
jobs from one year to the next. In Fiscal Year 2000-01, 62.1% of persons 
completing associate in science/college credit certificate programs were 
placed in higher wage jobs, compared to 60.1% in the prior year. 



Performance data is 
not comparable to 
standards for adult 
vocational certificate 
programs 



Due to the planned phase-in of performance funding, the Fiscal Year 
2000-01 performance data for adult vocational certificate programs is not 
comparable to prior year data or the standards. 10 As performance 
funding was being implemented, the department changed its method of 
measuring completions for these programs. Fiscal Year 2001-02 will be 
the first year in which performance data for adult vocational certificate 
programs will have been measured consistently for two years so that 
performance comparisons can be made. 



For Fiscal Year 2000-01, the number of adult vocational certificate program 
completers placed appears to increase compared to the prior year in all 
three wage categories, but this is actually due to the change in how 
completions are counted. For the time period evaluated by Fiscal Year 



10 Workforce development funding is partially based on program outcomes (completions and 
placements). At least 15% of the funds for workforce development education programs are 
distributed based on performance outcomes. The remaining funding is based on the prior year's 
funding levels. The Legislature created the performance funding system in Fiscal Year 1997-98. 
However, as discussed in a prior OPPAGA report, Fiscal Year 1999-00 was the first year in which data 
was available to allocate funds based on program outcomes. See Program Review, Workforce 
Development Education Program, Florida Department of Education, OPPAGA Report No. 01-56. 
November 2001 . 
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1999-00 measures, the department calculated "completions" based on 
students who had completed an entire program. 11 Beginning in Fiscal 
Year 2000-01, department data reflects the implementation of counting 
partial completions or Occupational Completion Points (OCPs). 12 A 
student may complete only part of a program, such as a certain number of 
courses, and the institution the student is attending receives credit for a 
completion point. The standard for Fiscal Year 2000-01 was developed 
when completions were counted using the prior method, and thus it is 
not comparable to the performance data for the year. Although the 
measure had not been reworded to reflect the implementation of OCPs, 
this change in methodology meets the intent of performance funding. 



The percentage of adult vocational certificate program completers placed 
appears to decrease over time and not meet the standards. However, this 
actually is due also to the implementation of OCPs. The standard for 
Fiscal Year 2000-01 and the Fiscal Year 1999-00 performance data reflect a 
different method of calculating completions and funding placements than 
was used for the Fiscal Year 2000-01 performance calculation. Because 
OCPs were not yet implemented as of the time period evaluated by Fiscal 
Year 1999-00 measures, students only "completed" and were available for 
placement one time. Beginning in the time period evaluated by Fiscal 
Year 2000-01 measures, the program does not fund placements if a 
student had been previously funded for a placement for the same 
program. This change increased the portion of students who are counted 
as not placed, decreasing the percentages of students placed in all 
categories. 



Lack of baseline data 
led to establishing an 
unrealistic standard for 
Adult Education 
Programs 



The planned phase-in of performance funding also affected performance 
evaluation for adult education programs. Due to changes in how adult 
education placements are calculated for performance funding purposes, 
the Department of Education did not have data to calculate the number of 
adult education, including English as a Second Language, and adult 
secondary education completion point completers who are found 
employed or continuing their education for Fiscal Year 1999-00 or prior 
years. As a result, the standard that was established for Fiscal Year 

2000- 01 was based on what was essentially an educated guess. The 
implementing bill for the Fiscal Year 2000-01 General Appropriations Act 
directed the department to establish this standard in its Fiscal Year 

2001- 02 Legislative Budget Request. Department documents show that 
the standard was established at 57,344. When data became available to 



11 Due to data collection methods, the Workforce Development Program's measures lag behind the 
time period being reported. For example. Fiscal Year 1999-00 measures represent persons who 
completed programs in 1997-98 and were placed as of 1998-99. Fiscal Year 2000-01 measures represent 
persons who completed programs in Fiscal Year 1998-99 and received job placements as of Fiscal Year 
1999-00. 

12 Occupational completion points were developed to identify benchmarks of student 
accomplishment. These are employability points in a program, permitting a student to leave before 
completing the program. 
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determine that performance results for Fiscal Year 2000-01 were 42,218 
completers placed, we concluded that the standard was unrealistic. 



The department has not 
completed data 
collection needed to 
report on measures 
relating to national 
accreditation 



The program has three additional measures relating to national 
accreditation, but the department has not completed data collection 
needed to establish baselines for these measures. Despite being required 
by the Fiscal Year 2000-01 and 2001-02 general appropriations acts to 
collect this data, the department did not begin a data collection effort 
until fall 2001. Department administrators initially attributed the lack of 
data collection to not having a reliable data source. However, upon 
further consideration, they determined that this information could be 
easily obtained through a survey of program administrators from 
community colleges and school districts. Department staff is in the 
process of conducting this survey effort and expects results by mid-June 
2002. 



The validity and reliability of the pmgram's performance 
data has not been established 



The Department of Education's inspector general is required to determine 
the validity of the measures and the accuracy of the associated data for 
each legislative performance measure; however, this determination has 
not been completed for the program's measures. 13 Consequently, our 
discussion of performance based on the program's legislative measures is 
provided without assurance that the measures are valid or that the 
associated data is accurate. 



The department 
inspector general has 
not determined the 
reliability of pmgram 
performance data 



Due to resource constraints and lack of available data, the inspector 
general has not reviewed source documentation and documentation 
processes to determine the reliability of the program's performance data. 
The Office of Inspector General reviewed some documentation relating to 
the department's Fiscal Year 2000-01 performance measures as reported in 
the agency' s Long-Range Program Plan for Fiscal Years 2002-2007, but this 
effort did not include review of source documentation and 
documentation processes for this program. The inspector general made 
recommendations to the department due to a lack of program data 
availability at the time of review. 



A review by the Florida Office of the Auditor General noted that 
improvements are needed in the reliability and consistency of student 
data from community colleges and school districts that is the original 
source data for some of the program's legislative performance measures. 



13 Required by s. 20.055, Florida Statutes. 

14 Operational Audit of the Florida Workforce Development Education Fund Appropriation for the 
Fiscal Year Ending June 30, 2001, Auditor General Report No. 01-136, March 2001. 



o 

ERLC 



18 



27 



Program Performance 



The Auditor General reviewed the accuracy of student data as part of an 
operational audit of the Workforce Development Education Fund 
appropriation. The Auditor General's tests disclosed a number of 
questioned practices and exceptions regarding the data for student 
completions and targeted student populations. The Auditor General also 
found that the current system provides data that is difficult to verify and 
allows varying interpretations in counting completions and in 
maintaining the underlying documentation for completions and targeted 
students. 

The inspector general plans to implement a data validation process over a 
three-year cycle. Department performance measures will be selected for 
validation based on a risk-assessment. The first review is scheduled to 
take place during the current fiscal Year (Fiscal Year 2001-02). 

Associate in science degree programs had low 
completion rates, but career and technical education 
pmgrams resulted in higher earnings forpmgram 
completers 

To supplement the limited information provided by the program's 
legislative performance measures, we further evaluated the program's 
effectiveness by (1) analyzing participant completion rates for career and 
technical education programs and (2) assessing whether these programs 
are providing individuals with the training and tools needed to improve 
their earnings. These analyses showed that completion rates were low for 
associate in science degree programs when compared to other types of 
programs, but program completers were better off financially for having 
participated in career and technical education programs. 

Completion rates tow Completion rates measure the extent to which individuals actually finish 
for associate in science a program. 15 Thus, completion rates provide an indicator of the success 
degree programs of community colleges and school districts in assisting students to achieve 

economic self-sufficiency. 

Our analysis of the 124,204 students who entered career and technical 
education programs in 1995-96 showed that completion rates vary greatly 



15 In this context, we are using the term "completion rate" to mean full program completion, not 
completion of individual Occupational Completion Points that make up a program. 
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Various factors 
contribute to low 
completion rates 



by type of program. 16 (See Exhibit 6.) Completion rates are highest for 
students who entered college credit certificate programs and lowest for 
students who entered associate in science degree programs. On average 
for all programs, 27.2% of the students completed a workforce 
development education program, but the completion rates were 17% for 
students in associate in science degree programs. 



Exhibit 6 

Completion Rates Low in Associate in Science Degree Programs 



0 Completed a Workforce Development Education Program 
■ Did Not Complete a Program 



to Associate in Science Degree (25,292) 
© 

OJ 

JD 

o 

o 

>* Adult Vocational Certificate (15,285) 



E 

E 

o 

o 



College Credit Certificate (1,974) 



Jc £ Adult Vocational Certificate (81,653) 

O M 

CO ~ 



17 . 0 % 




83 . 0 % 



72 . 2 % 



Note: Apprenticeship programs are reported with adult vocational certificate programs. 

Source: OPPAGA analysis of Department of Education data on students who entered workforce 
development education programs in 1995-96. 



Several factors contribute to low completion rates for associate in science 
degree programs. A major factor affecting completion rates is program 
length. Associate in science degree programs take two years to complete. 
In comparison, certificate programs typically take one year to complete, 
depending on the occupation. 17 Associate in science degree programs 
thus require a larger commitment of time and money from students and 
the workforce system. 



16 We examined a cohort of all first-time students who entered a postsecondary school district or 
community college workforce development education program in the 1995-96 school year. We chose 
this time period to increase the probability that the students would finish their programs and to 
provide at least four quarters of post-completion income to compare to their earnings prior to 
entrance. Although workforce development programs can be completed within two years, many 
students attend school on a part-time basis and require more than two years to finish their programs. 
The 1995-96 cohort was the most recent group of students that would have four quarters of income 
data after program completion. 

17 Program lengths assume full-time attendance. 
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Other factors also contribute to low completion rates. Students may 
decide they have chosen an occupation that is not suitable for them, or 
they may decide to enter the labor market before completing a program. 
According to program administrators, when labor market demands are 
high, employers recruit many students before they complete programs. 
Also, students may discontinue programs for personal reasons, such as 
lack of interest or the need to work more hours to support a family. For 
the group of student records we analyzed, the median age of persons in 
community college career and technical education programs was 31 and 
the median age for school district career and technical education 
programs was 33. 

Eamings increase for Our analyses also showed that workforce development programs result 
program completers in increased earnings for program participants. We evaluated 

performance outcomes for students who entered workforce development 
education program in 1995-96. 18 This analysis showed that eamings 
increased for all groups enrolled, with greater eamings increases for 
individuals who completed a program. (See Exhibit 7 and Appendix F.) 

Exhibit 7 

Program Completers Earn More Than Non-Completers in 

All Career and Technical Education Programs (Annual Median Eamings) 




18 The most recent wage data available at the time we conducted this analysis was for the first quarter 
of 2001. In order to incorporate the most recent wage data, we compiled annual data using data for 
the second, third, and fourth quarter for a given year plus data for the first quarter for the following 
year. For example, wage data for 2000-01 consists of data for the second, third, and fourth quarter of 
2000 and the first quarter of 2001. 
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Note: We used the second, third, fourth quarters and the first quarter from the following year as the four 
quarters making up a year. See Appendix F for the number of completers and non-completers each year. 
Source: OPPAGA analysis of Department of Education data on students who entered workforce 
development education programs in 1995-96. 
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Many factors other than completing a workforce development education 
program can influence how much a person earns. For example, programs 
that attract people who have more work experience and higher wages 
prior to entering the program will appear to be more successful when 
comparing post-completion earnings to those of programs that attract less 
experienced and skilled people. Program selection also varies by certain 
demographic characteristics. For example, more women enter nursing 
and childcare programs and more men go into law enforcement and fire 
fighting. 

When comparing the performance of workforce development education 
programs or delivery systems, it is important to consider these other 
factors. Performance differences may be due to the types of people 
entering the programs and not to the program. The best comparison 
would compare students with the same characteristics across each 
program. 

As shown in Exhibit 8, when considering student characteristics that affect 
earnings, students completing a workforce development education 
program can expect to earn more than those that do not. The predicted 
improvement in annual earnings between completers and non- 
completers across workforce development education programs ranges 
from $1,402 to $3,776. 



Exhibit8 

Predicted Improvement in Annual Earnings of Completers Compared to 
Non-Completers of Career and Technical Education Programs 1 



Community 
Colleges Associate 
in Science Degree 

Community College 
Adult Vocational 
Certificate 

Community 
Colleges College 
Credit Certificate 

School Districts 
Adult Vocational 
Certificate 




1 We used a statistical technique, least squares regression, to account for the effect of other factors 
(earnings prior to entering program (1994-95), sex, age, race, and regional wage differences) on earnings 
(second, third, fourth quarter 2000 and first quarter 2001). 

Source: OPPAGA analysis of Department of Education data on students who entered workforce 
development education programs in 1995-96 
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Revisions to measures needed to provide valid 
conclusions about performance 

The Workforce Development Education Program's legislative 
performance measures as shown in Exhibit 5 will provide some useful 
information once the effects of the planned phase-in of performance 
funding no longer prevent performance comparison across years. 
However, these measures are not sufficient to draw conclusions about 
program performance. The measures should emphasize a higher level of 
post-completion earnings by increasing the Level HI wage level and 
eliminating Levels II and I as performance targets. The measures do not 
include completion rates or adequate outcome measures for adult basic 
education programs. 

The Level Hi wage Although the purpose of this program is to help students attain those 

target Should be skills that enable them to become or remain economically self-sufficient, 

increased the program's measures do not sufficiently stress achievement of this 

goal. The measures establish the Level IE or "high wage" placement as 
the highest tier of performance, meaning that an entry salary level of 
$18,720 annually ($9 an hour) is the highest salary goal achievement being 
measured. However, entry level earnings of $18,720 annually are not 
usually considered high wages by the general public, especially for a 
person supporting a family. "High wage" is a relative term used to 
describe a level of wages considered an improvement for a segment of the 
population that may have been marginally employed (making minimum 
wage and/or not working full-time) when entering these programs. 

The Workforce Estimating Conference has recognized that the $9 
threshold should be raised and opted to change the Fiscal Year 2002-03 
high wage entry-level job earnings threshold to $10.05 ($20,904 annually), 
with an average wage of $13.86 ($28,829 annually). The performance 
target for high wage placements should thus be increased to at least 
$10.05 an hour ($5,226 per quarter). This performance target should be 
increased gradually over time to reflect increases in expected salary levels 
for the types of occupations for which program participants are receiving 
training and education. The program should evaluate the average wage 
for program completers and set a goal to increase the percentage 
achieving the "high wage" level. 



Measures Should not As currently implemented, the program's measures detract from focusing 

include Levels / and II on improving salary levels by including achievement of Level I and Level 

placements n wage levels. A Level I placement is a salary level that could be achieved 

in the absence of career and technical education and is contrary to the 
program's intent. This level of placement means the participant has some 
employment but the income is below $7.50 an hour, is in the military, or is 
continuing education in a vocational certificate program. Any of these 
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Measures should 
include completion 
rates 



Additional outcome 
measure needed for 
adult basic education 



outcomes could be achieved without attending a career and technical 
education program. A minimum wage job earning $5.15 an hour would 
qualify as a Level I placement. According to s. 239.101(4), Florida Statutes, 
. . the purpose of career education is to enable students to attain those 
skills that enable them to become or to remain economically self- 
sufficient. Consequently, the Legislature finds that vocational programs 
which lead to minimum wage employment should be minimized and 
should be conducted only with specific justification." 

Although not a minimum wage placement, a Level II placement (at least 
$7.50 an hour or $15,600 annually) is just equal to or below a living wage. 
A living wage is defined as the amount of money required to sustain a 
family of three above the poverty line. Estimates of the living wage range 
from $6.10 to $12 per hour, with $6.10 to $8 ($12,688 to $16,400 annually) 
commonly dted. Thus, a Level II wage level target is barely adequate for 
economic self-sufficiency. 

The program's measures would prove a more meaningful picture of 
program effectiveness by removing the Level 1 and Level H placements as 
performance targets and establishing one target wage level. This level 
should be the high wage placement (currently Level HI). 

The program's performance measures should also include completion 
rates. These rates are important because only a small portion of 
participants actually complete some programs, especially associate in 
science degree programs, and attention should be focused on improving 
completion rates to the extent possible. 

The program's measures should also include an additional outcome 
measure for adult basic education. These activities are currently 
addressed by the measure number of adult basic education, including 
English as a Second Language, and adult secondary education completion 
point completers who are found employed or continuing their education. 
Although this is useful information, the number of completers placed 
does not provide sufficient information by itself to evaluate effectiveness. 
Increases in the number of completers placed may only reflect increases in 
enrollment rather than improvements in the extent to which completers 
obtain employment or continue their educations. To put this number into 
proper context, the program's measures should include the percentage of 
completers of adult basic education who obtain employment or continue 
their education. 

OPPAGA's proposal for revisions to the program's measures is 
summarized in Exhibit 9. We have included a rewording of measures for 
adult vocational certificate programs to reflect the implementation of 
Occupational Completion Points, as has been proposed in the pending 
general appropriations bills for Fiscal Year 2002-03. 
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Exhibit 9 

OPPAGA Recommendations for Workforce Development Education 
Program Performance Measures 



Recommended .Performance Measures . l-W •u.cvk f < >i 


Adult Vocational 
Certificate 


Number and percentage of adult vocational certificate program completion 
point completers who are found employed at $5,226 or more per quarter 
($10.05/ hour full-time) 




Number and percentage of participants who fully complete adult vocational 
certificate programs 


College Credit 
Certificate/Associate in 
Science Degree 


Number and percentage of college credit certificate program completers 
who are found employed at $5,226 or more per quarter ($1 0.05/ hour full- 
time) 


Number and percentage of participants who fully complete college credit 
certificate/ associate in science degree programs 


Adult Basic Education 
Programs 


Number and percentage of adult basic education, including English as a 
Second Language, and adult secondary education completion point 
completers who are found employed or continuing their education 


Nationally Recognized 
Accrediting Standards 


Number and percentage of workforce development programs which meet or 
exceed nationally recognized accrediting standards for those programs 
which teach a subject matter for which there is a nationally recognized 
accrediting body 




Number and percentage of students attending workforce development 
programs which meet or exceed nationally recognized accrediting standards 




Number and percentage of students completing workforce development 
programs which meet or exceed nationally recognized accrediting standards 



Source: OPPAGA analysis. 



Conclusions and Recommendations 

The program's performance in meeting Fiscal Year 2000-01 legislative 
performance standards can be evaluated only for associate in science 
degree and college credit certificate programs. The program met most 
standards for the number and percentage of associate in science degree 
and college credit certificate program completers placed in jobs or 
continuing their education. We could not evaluate the program's 
performance for the remaining legislative measures for three reasons: 

(1) performance data for the measures assessing adult vocational 
certificate programs is not comparable to data for prior years due to the 
planned phase-in of performance funding over time, (2) baseline data to 
establish reasonable standards has only recently become available for the 
measure assessing adult education programs, and (3) the department has 
not completed data collection needed to establish performance and 
standards for the measures relating to national accreditation. 
Supplemental performance analysis shows that associate in science 
degree programs have low completion rates, but career and technical 
education programs result in higher earnings for program completers. 
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The program's legislative performance measures will provide some useful 
information once the effects of the planned phase-in of performance 
funding no longer prevent performance comparison across years. 
However, these measures are not sufficient to draw conclusions about 
performance because they do not adequately emphasize higher levels of 
post-completion earnings, and do not include completion rates or 
adequate outcome measures for adult basic education programs. To 
improve the usefulness of the program's legislative performance measures 
in evaluating program performance, the Legislature should consider 
revising the program's measures to focus on higher levels of post- 
completion earnings, include completion rates for career and technical 
education programs, and include an additional outcome measure for 
adult basic education. OPPAGA's suggestions for improving the 
program's measures are illustrated in Exhibit 9. 



Appendix A 

Requirements for Justification Reviews 



Section 11.513(3), Florida Statutes, provides that OPPAGA Program 
Evaluation and Justification Reviews shall address nine issue areas. Our 
conclusions on these issues as they relate to the Workforce Development 
Education Program are summarized in Table A-l. 

Table A-1 

Summary of the Program Evaluation and Justification Review of 
the Workforce Development Education Program 



L Issue A'C'v 


OPR^;GoncluSi6ns^ &-?■” 


The identifiable cost of the program 


The program is funded though general revenue and trust funds. The program was 
appropriated $695,740,368 in general revenue and $100,602,397 in trust funds for 
Fiscal Year 2001-02. 


The specific purpose of the program, 
as well as the specific public benefit 
derived therefrom 


The Workforce Development Education Program provides career and technical 
education and other educational services to help students attain those skills that 
enable them to become or remain economically self-sufficient. The program provides 
career and technical education, adult education, and continuing workforce education 
program services. In Fiscal Year 2000-01, school district programs served 643,901 
students and community colleges served 360,588 students in workforce programs. 

The program provides a public benefit by providing training designed to meet state 
and local workforce needs and helping Florida compete in a global economy. Building 
a broadly based, highly skilled, more productive workforce contributes to Florida’s 
economic prosperity. The program also helps the state avoid costs by reducing 
dependence upon public assistance and reducing the incidence of incarceration in the 
state's correctional system. 


Progress towards achieving the 
outputs and outcomes associated with 
the program 


The program’s performance in meeting Fiscal Year 2000-01 legislative performance 
standards can only be evaluated for associate in science degree and college credit 
certificate programs. The program met most standards for the number and 
percentage of associate in science degree and college credit certificate program 
completers placed in jobs or continuing their education. 

Due to program failure to collect data for national accreditation measures and year-to- 
year changes in data resulting from the planned phase-in of the performance funding 
system, we could not determine the program’s performance for the remaining 
measures. Phasing in performance funding resulted in changes from one year to the 
next in how completions and placements were counted for adult vocational education 
programs. Lack of baseline data led to establishing an unrealistic standard for adult 
education programs. The program has not completed data collection needed to report 
on the extent to which programs meet national accreditation standards and students 
attend programs meeting the accreditation standards. 


An explanation of circumstances 
contributing to the state agency's 
ability to achieve, not achieve, or 
exceed its projected outputs and 


Although it would appear initially that adult vocational certificate programs 
experienced increases in the number and percentage of students placed in job 
categories, the planned phase-in of performance funding led to data problems with 
these measures. Due to changes in how completions and placements are counted for 
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■ .'■■OPPAGA Conclusions ? < •. . v -f 


outcomes, as defined in s. 216.011, 
F.S., associated with the program 


these programs, Fiscal Year 2000-01 performance data is not comparable to prior 
year data or standards. 


Alternative courses of action that 
would result in administering the 
program more efficiently and 
effectively 


Recommendations specifically related to Adult Education Programs will be discussed 
in an upcoming OPPAGA report. 

Recommendations specifically related to Apprenticeship Programs will be discussed 
in an upcoming OPPAGA report. 


The consequences of discontinuing 
the program 


If state funding for workforce development education programs were discontinued, 
the likely short-term impact would be a reduction in the availability of career and 
technical education and adult education programs for the general public and higher 
costs for program participants. The long-term effect of fewer of these programs 
would be fewer trained persons for Florida’s job market, an increased reliance on 
public assistance, and a higher state incarceration rate. Although private institutions 
and local workforce investment boards might take over some of the program’s 
responsibilities, neither would be in a position to fully meet the service demands 
currently being met by the Workforce Development Education Program. 


Determination as to public policy, 
which may include recommendations 
as to whether it would be sound public 
policy to continue or discontinue 
funding the program, in whole or in 
part, in the existing manner 


The public benefits derived from the program’s workforce development education 
services indicate that it is sound public policy to continue funding the program. 
General revenue funding is appropriate for this program because the program 
primarily benefits general taxpayers. These benefits include contributing to the state’s 
economic growth and well-being, and reducing crime rates and participants' reliance 
on public assistance programs. Also, for many workforce development education 
programs, participants have no means to pay the cost, and it is contrary to the 
program’s premise to require them to do so. 


Whether the information reported as 
part of the state’s performance-based 
program budgeting system has 
relevance and utility for evaluation of 
the program 


Once the effects of the planned phase-in of performance funding no longer prevent 
performance comparison across years, the program’s legislative performance 
measures will provide some useful information. However, revisions to these 
measures are needed to better evaluate program performance. The measures do not 
sufficiently emphasize higher levels of post-completion earnings and do not include 
completion rates or adequate outcome measures for adult basic education programs. 


Whether state agency management 
has established control systems 
sufficient to ensure performance data 
are maintained and supported by 
agency records and accurately 
presented in agency performance 
reports 


Due to resource constraints and lack of available data, the Department of Education's 
inspector general has not reviewed source documentation and documentation 
processes to determine the reliability of the program's performance data. The Office 
of Inspector General reviewed some documentation relating to the departments Fiscal 
Year 2000-01 performance measures as reported in the agency’s Long-Range 
Program Plan for Fiscal Years 2002-2007, but this effort did not include review of 
source documentation and documentation processes for this program. The inspector 
general made recommendations to the department due to a lack of program data 
availability at the time of review. The inspector general plans to implement a data 
validation process over a three-year cycle beginning in Fiscal Year 2001 -02. 


Source: OPPAGA analysis. 
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Activities of the Three Divisions Assigned 
Workforce Development Education Program 
Responsibilities 



Division of Workforce Development The Division of Workforce 
Development provides oversight for all public postsecondary workforce 
development education programs. The division's oversight activities 
include developing designs for instructional programs, developing 
strategies to increase student productivity, and providing professional 
development activities for school district and community college 
workforce development programs. The Division of Workforce 
Development provides administrative direction to K-12 as well as 
postsecondary programs. 19 

Division of Community Colleges. The Division of Community Colleges 
provides additional oversight for community college workforce 
development programs. The oversight responsibilities are similar to those 
provided by the Division of Workforce Development, i.e., developing 
designs for instructional programs and strategies to increase student 
productivity. Community colleges have formed numerous partnerships 
with local businesses and industry, and many of these are designed to 
meet the growing demands of the technology industry. 

Division of Technology. In the Division of Technology, the Workforce 
Education and Outcome Information Services (WEOIS) provides planning 
and information services, leadership and technical assistance at the local, 
state, and national levels of workforce development. WEOIS is 
responsible for maintaining the Workforce Education Information System, 
and collects, analyzes, and disseminates enrollment and follow-up data 
on workforce education programs. WEOIS provides technical assistance 
and maintains a data base handbook for local and state workforce 
education staff. 

WEOIS works with legislative staff to determine annual allocations for 
community colleges and school districts based on performance outcomes 
(completions and placements). WEOIS also plays a key role in the 
Workforce Estimating Conference by developing forecasts of job 
openings, employment, and earnings. The occupational forecasts are 



19 OPPAGA reviewed the K-12 programs in Justification Review, Kindergarten Through Twelfth 
Grade Public Education Program, OPPAGA Report No. 01-22. April 2001 . 
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used to identify vocational programs needed to meet statewide and local 
workforce demands. WEOIS distributes this data to local community 
colleges and school districts where it is used as a planning tool in 
designing workforce development education programs. WEOIS includes 
the Florida Education and Training Placement Information Program 
(FETPIP), a data collection system that obtains follow-up data on former 
students. The placement information includes employment, continuing 
post-secondary education, military, public assistance participation, and 
incarceration data. 
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Career and Technical Education Programs 



Career and technical education programs prepare students to enter 
specific occupations. For Fiscal Year 2000-01, community colleges served 
120,981 students and school districts served 88,355 students in career and 
technical education programs. The four types of career and technical 
education programs provided by the Workforce Development Education 
Program are described below. 

■ Associate in science degree. Associate in science degree programs 
provide courses that lead to employment in a specified occupation. 
Students complete core courses and the general education 
requirements, and may transfer to a four-year institution or enter the 
job market. For example, the business administration degree requires 
24 hours of general education, such as math, economics, and 
composition, and 40 hours of core courses such as finance, 
management, accounting, and marketing. Other examples of 
associate in science degree program are office systems technology, 
electronic engineering technology, and criminal justice technology. 

For Fiscal Year 2000-01, community colleges served 78,916 students in 
associate in science degree programs. 

■ College credit certificate. College credit certificate programs prepare 
students for specific occupations and may not have the general 
education requirements of associate in science degree programs. 
Examples of college credit certificate programs include accounting 
applications, business data processing, paramedic, radiation therapy 
specialist, and small business management. For Fiscal Year 2000-01, 
community colleges served 9,427 students in college credit certificate 
programs. 

■ Adult vocational certificate. Adult vocational certificate programs 
provide courses that lead to employment in a specific occupation and 
terminate with the student earning either a certificate or an applied 
technology diploma. Examples of adult vocational certificate 
programs are administrative assistant, automotive technology, 
culinary arts, communication electronics, and network support 
services. For Fiscal Year 2000-01, community colleges served 32,638 
students in adult vocational certificate programs. School districts 
served 88,355 students, of which 9,952 were enrolled in apprenticeship 
programs. 
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■ Apprenticeship and pre-apprenticeship. Apprenticeship and pre- 
apprenticeship programs combine on-the-job training and related 
instruction in which workers learn the practical and theoretical 
aspects of highly skilled occupations. Individuals work full-time and 
attend school in the evenings. A major advantage of apprenticeship 
programs is that students earn wages immediately and have periodic 
wage increases as they complete portions of the program. 
Apprenticeship programs have been the main pathway to entering a 
skilled trade such as an electrician, plumber, and brick mason. For 
Fiscal Year 2000-01, school districts served 9,952 students in 
apprenticeship programs. 20 



20 The Workforce Development Education Program did not have information available to readily 
distinguish apprenticeship students from adult vocational certificate students for community colleges. 
Apprenticeship students are included with the figures given for adult vocational certificate programs. 
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Adult Education Programs 

Adult education programs provide courses to improve the employability 
of adults with poor literacy or English language skills, those who need to 
complete a high school education, and adults with disabilities. For Fiscal 
Year 2000-01, community colleges served 64,439 students and school 
districts served 458,795 students in adult education programs. The four 
types of adult education programs provided by the Workforce 
Development Education Programs are described below. 

■ Adult basic education. Adult basic education is beginning literacy 
through the eighth-grade level in mathematics, reading, language, 
and workforce readiness skills. For Fiscal Year 2000-01, community 
colleges served 36,373 students and school districts served 96,076 
students in adult basic education programs. 

■ Adult secondary education. Adult secondary education is instruction 
from the ninth grade level through high school. It includes General 
Educational Development (GED), which is instruction from ninth 
grade through high school that leads to a GED diploma. For Fiscal 
Year 2000-01, community colleges served 21,324 students and school 
districts served 124,871 students in adult secondary education 
programs. 

■ English for speakers of other languages. English for speakers of other 
languages education is instruction in English reading, writing, 
speaking, and listening skills needed to succeed in the workplace. For 
Fiscal Year 2000-01, school districts served 156,471 students in English 
for speakers of other languages programs. 21 

■ Other adult programs. Other adult programs include programs such 
as vocational preparatory education, workplace readiness training, 
and training for adults with disabilities. Vocational preparatory 
education is instruction designed for the student to attain academic 
and workforce readiness skills so that they may pursue certificate 
career education or higher-level career education. Workplace 
readiness training is instruction designed to develop skills necessary 
to function in the workplace (timeliness, proper attire, interview 
skills). Training for adults with disabilities is instruction in literacy, 
work-related behaviors, and daily living skills to help disabled adults 
succeed in the workplace. For Fiscal Year 2000-01, community 
colleges served 6,742 students and school districts served 81,377 
students in these programs. 



21 Community colleges do not separate counts for students served by English for speakers of other 
languages programs. 
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Continuing Workforce Education Programs 

Continuing workforce education programs are short-term training 
programs that are typically business-sponsored and designed to upgrade 
skills for individuals who are already employed. For Fiscal Year 2000-01, 
community colleges served 175,168 students and school districts served 
96,751 students in continuing workforce education programs. The two 
types of continuing workforce education programs provided by the 
Workforce Development Education Programs are described below. 22 

■ Short-term skills training. These training sessions are designed to 
upgrade skills and are typically employer-sponsored. The programs 
may be provided at the place of business or at community colleges or 
school districts. Community colleges and school districts may create 
business institutes as a means to provide and advertise these services. 
The programs are usually one-half day to two-day courses designed to 
upgrade skills in areas such as computer technology, human resource 
management, communication, and customer satisfaction. 

■ Continuing education requirements for occupational licenses. These 
programs provide training courses for individuals who are currently 
employed and need continuing workforce education to maintain 
certifications. Examples of continuing education programs are those 
provided for individuals who are licensed in areas such as real estate 
or insurance and are required to keep their licenses current through 
continuing education. 



22 Information was not readily available to determine the number of students served in each of the 
two major types of continuing workforce education programs. 
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Earnings Over Time for Students in 
1 995-96 Cohort 



To provide a more complete picture of the performance of the Workforce 
Development Education Program, we examined the cohort of all first- 
time-in-program students who entered a postsecondary public school 
district or community college workforce development education program 
in 1995-96. Data was received from both community colleges and school 
districts. The 1995-96 cohort of students was chosen so as to maximize the 
probability that the students would have had time to finish their 
programs and have at least four quarters of post-completion income to 
compare to their income prior to entrance. The 1995-96 cohort was the 
most recent group of entrants for which we could have enough post- 
completion data for our analysis. The median annual earnings for these 
students are shown in Table F-l. 



Table F-1 

Earnings Over Time for Students Who Entered Postsecondary Public School District or 
Community College Workforce Development Education Programs in 1995-96 



Workforce Development 
Education Programs 


1994-95 


1995-96 


1996-97 


1997-98 


1998-99 


1999-00 


2000-01 


Associate in Science Degree 


f * a \ 


_ V.v” 


■_ ~'A . **■ . 




; H : : . * 




- r.-4 

V:.. , -I 


Completers 


Mean 


$14,162 


$13,459 


$14,890 


$16,614 


$20,364 


$24,744 


$29,259 




Median 


" $11, 629 


V $9,992 


. $11,156 


$13,294 


$18,128 


: $22,882 


$27,575 




N 


1,322 


1,383 


1,373 


1,548 


1,641 


1,644 


1,628 


Non-Completers 


Mean 


$11,630 


$11,812 


$13,219 


$15,618 


$17,529 


$19,422 


$21,032 




Median 


$8,112 


$8,510 


$10,488 


$13,420 


$15,561 


$17,466 


' $19,043 




N 


14,650 


15,920 


16,140 


16,732 


16,209 


15,752 


15,321 


^oUege Credit Vocational Certificate 


. v 7 * 






■vTV-"' 1 








Completers 


Mean 


$15,442 


$15,800 


$17,455 


$21^641 


$24,151 


$26,899 


$29,811 




Median 


$12,436 


$11,903 


$14,243 


$18,431 


$21,707 


$24,301 


$27,008 




N 


630 


652 


686 


711 


707 


693 


668 


Non-Completers 


Mean 


$14,069 


$14,142 


$16,139 


$18,840 


$20,533 


$22,921 


$24,182 




Median 


$10,089 


$10,174 


$13,489 


$16,026 


$17,934 


$20,146 


$21,168 




N 


727 


750 


764 


785 


768 


753 


748 




36 



45 



Appendix F 



Workforce Development 
Education Programs 


1994-95 1995-96 1996-97 1997-98 1998-99 1999-00 2000-01 


| 

i 


mwm 








Completers 


Mean 


$12,352 


$12,075 


$15,853 $19,274 $21,523 $23,788 $25,361 




Median 


' 7 $9,985 


$10,619 


$24*324 $25,653 




N 


3,211 


3,862 


4,044 4,098 3,937 3,808 3,717 


Non-Completers 


Mean 


$11,303 


$10,672 


$12,441 $14,979 $16,776 $18,410 $19,716 




Median 


• '1S8.230 


$7,595..; 


' $9,816';. $12,81 8 : $ 1 4, 631: $1 6, 1 60^ $1 7,157 




N 


5,761 


6,637 


6,915 7,147 6,974 6,714 6,574 










Completers 


Mean 


$9,780 


$9,144 


$11,537 $14,425 $16,476 $18,241 $19,660 




iMediah l 




{ $6,514 \ 


| $9,636 ~ : $1 3,025 $15^267 ^6,8^ 




N 


10,275 


11,711 


12,918 13,187 12,796 12,347 11,959 


Non-Completers 


Mean 


$9,868 


$9,174 


$9,925 $12,657 $14,614 $16,133 $17,315 




iMedituij 


1^6,584^ 


^5,952^ 


y ^7,1 4 q ^/-:-$1 p,606 ^#$i4,Q7pl" ig^i'5;20gi 




N 


28,629 


32,218 


33,798 35,192 33,931 32,831 31,827 



Note: We used the second, third, fourth quarters and the first quarter from the following year as the four quarters making up a year. 
Source: OPPAGA analysis of Department of Education data on students who entered workforce development education programs 
in 1995-96. 
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Response from the 
Department of Education 

In accordance with the provisions of s. 11.51(5), Florida Statutes, a draft of 
our report was submitted to the Commissioner of Education and the 
Secretary of the Department of Education for their review and response. 
Their written response is reprinted herein beginning on page 39. 
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F. Philip Handy 
Chariman 

Linda J. Eads, Ed.D 

T. williard Fair 

Charles Patrick Garcia 

JULIA L. JOHNSON 



FLORIDA BOARD OF EDUCATION 



May 20, 2002 



Mr. John W. Turcotte, Director 
Office of Program Policy Analysis 
and Government Accountability 
1 1 1 West Madison Street, Room 312 
Claude Pepper Building 
Tallahassee, FL 32399-1475 



william L. Proctor, Ph.d 



Carolyn King Roberts 

Dear Mr. Turcotte: 



The Department of Education is appreciative of the hard work and 
thoroughness of the Justification Review of the Workforce Development 
Program. Attached is the Department's official response to this justification 
review. 



If you require additional information please feel free to contact 
Loretta Costin, Director, Division of Workforce Development. 

Sincerely, 



1st 1st 

Jim Horne Charlie Crist 

Secretary Commissioner 



Charlie Crist 

Commissioner 
Florida Department 
of education 



Attachment: Justification Review of 

Workforce Development Program 



MM/dsh 

325 w. Gaines Street 
SurTEl6l4 

TALLASSEE, FL 32399 \ : 

www.flboe.org 

(850) 201-7400 
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Department of Education Response to 
OPPAGA Justification Review of the 
Workforce Development Education Program 

The Department of Education appreciates the opportunity to comment on the OPPAGA 
Justification Review and the Department concurs with the statement that the program 
provides a public benefit and should continue. This program prepares over one million 
students for employment by providing career and technical education training or literacy 
instruction. 

The Department also concurs with the recommendations regarding the modifications to 
the Program Performance Measures. 
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